Dual purposes by design: exploring alignment between residents' and academic advisors' documents in a longitudinal program.

Longitudinal academic advising (AA) and coaching programs are increasingly implemented in competency based medical education (CBME) to help residents reflect and act on the voluminous assessment data they receive. Documents created by residents for purposes of reflection are often used for a second, summative purpose-to help competence committees make decisions-which may be problematic. Using inductive, thematic analysis we analyzed written comments generated by 21 resident-AA dyads in one large internal medicine program who met over a 2 year period to determine what residents write when asked to reflect, how this aligns with what the AAs report, and what changes occur over time (total 109 resident self-reflections and 105 AA reports). Residents commented more on their developing autonomy, progress and improvement than AAs, who commented far more on performance measures. Over time, residents' writing shifted away from intrinsic roles, patient care and improvement towards what AAs focused on, including getting EPAs (entrustable professional activities), studying and exams. For EPAs, the emphasis was on getting sufficient numbers rather than reflecting on what residents were learning. Our findings challenge the practice of dual-purposing documents, by questioning the blurring of formative and summative intent, the structure of forms and their multiple conflicting purposes, and assumptions about the advising relationship over time. Our study suggests a need to re-evaluate how reflective documents are used in CBME programs. Further research should explore whether and how documentation can best be used to support resident growth and development.

Modeling Diagnostic Expertise in Cases of Irreducible Uncertainty: The Decision-Aligned Response Model.

PURPOSE: Assessing expertise using psychometric models usually yields a measure of ability that is difficult to generalize to the complexity of diagnoses in clinical practice. However, using an item response modeling framework, it is possible to create a decision-aligned response model that captures a clinician's decision-making behavior on a continuous scale that fully represents competing diagnostic possibilities. In this proof-of-concept study, the authors demonstrate the necessary statistical conceptualization of this model using a specific electrocardiogram (ECG) example. METHOD: The authors collected a range of ECGs with elevated ST segments due to either ST-elevation myocardial infarction (STEMI) or pericarditis. Based on pilot data, 20 ECGs were chosen to represent a continuum from "definitely STEMI" to "definitely pericarditis," including intermediate cases in which the diagnosis was intentionally unclear. Emergency medicine and cardiology physicians rated these ECGs on a 5-point scale ("definitely STEMI" to "definitely pericarditis"). The authors analyzed these ratings using a graded response model showing the degree to which each participant could separate the ECGs along the diagnostic continuum. The authors compared these metrics with the discharge diagnoses noted on chart review. RESULTS: Thirty-seven participants rated the ECGs. As desired, the ECGs represented a range of phenotypes, including cases where participants were uncertain in their diagnosis. The response model showed that participants varied both in their propensity to diagnose one condition over another and in where they placed the thresholds between the 5 diagnostic categories. The most capable participants were able to meaningfully use all categories, with precise thresholds between categories. CONCLUSIONS: The authors present a decision-aligned response model that demonstrates the confusability of a particular ECG and the skill with which a clinician can distinguish 2 diagnoses along a continuum of confusability. These results have broad implications for testing and for learning to manage uncertainty in diagnosis.

Goal co-construction and dialogue in an internal medicine longitudinal coaching programme.

BACKGROUND: Longitudinal coaching in residency programmes is becoming commonplace and requires iterative and collaborative discussions between coach and resident, with the shared development of goals. However, little is known about how goal development unfolds within coaching conversations over time and the effects these conversations have. We therefore built on current coaching theory by analysing goal development dialogues within resident and faculty coaching relationships. METHODS: This was a qualitative study using interpretive description methodology. Eight internal medicine coach-resident dyads consented to audiotaping coaching meetings over a 1-year period. Transcripts from meetings and individual exit interviews were analysed thematically using goal co-construction as a sensitising concept. RESULTS: Two themes were developed: (i) The content of goals discussed in coaching meetings focused on how to be a resident, with little discussion around challenges in direct patient care, and (ii) co-construction mainly occurred in how to meet goals, rather than in prioritising goals or co-constructing new goals. CONCLUSIONS: In analysing goal development in the coach-resident relationships, conversations focused mainly around how to manage as a resident rather than how to improve direct patient care. This may be because academic coaching provides space separate from clinical work to focus on the stage-specific professional identity development of a resident. Going forward, focus should be on how to optimise longitudinal coaching conversations to ensure co-regulation and reflection on both clinical competencies and professional identity formation.

Supervising the senior medical resident: Entrusting the role, supporting the tasks.

INTRODUCTION: Postgraduate competency-based medical education has been implemented with programmatic assessment that relies on entrustment-based ratings. Yet, in less procedurally oriented specialties such as internal medicine, the relationship between entrustment and supervision remains unclear. We undertook the current study to address how internal medicine supervisors conceptualise entrusting senior medical residents while supervising them on the acute care wards. METHODS: Guided by constructivist grounded theory, we interviewed 19 physicians who regularly supervised senior internal medicine residents on inpatient wards at three Canadian universities. We developed a theoretical model through iterative cycles of data collection and analysis using a constant comparative process. RESULTS: On the internal medicine ward, the senior resident role is viewed as a fundamentally managerial and rudimentary version of the supervisor's role. Supervisors come to trust their residents in the senior role through an early 'hands-on' period of assessment followed by a gradual withdrawal of support to promote independence. When considering entrustment, supervisors focused on entrusting a particular scope of the senior resident role as opposed to entrustment of individual tasks. Irrespective of the scope of the role that was entrusted, supervisors at times stepped in and stepped back to support specific tasks. CONCLUSION: Supervisors' stepping in and stepping back to support individual tasks on the acute care ward has an inconsistent relationship to their entrustment of the resident with a particular scope of the senior resident role. In this context, entrustment-based assessment would need to capture more of the holistic perspective of the supervisor's entrustment of the senior resident role. Understanding the dance of supervision, from relatively static overall support of the resident in their role, to fluidly stepping in and out for specific patient care tasks, allows us insight into the affordances of the supervisory relationship and how it may be leveraged for assessment.

A qualitative study of medical students' perceptions of resident feedback.

CONTEXT: Residents play a pivotal role in medical students' clinical education. From a feedback lens, the near-peer relationship between student and resident holds the potential to foster an educational alliance that could influence learning. We undertook the current qualitative study to explore medical students' perceptions of feedback experiences with residents, addressing when, how and why (and conversely when not and why not) resident feedback plays a role in their clinical education. METHODS: Our methodology was qualitative interpretive description, informed by phenomenology. We conducted 24 semi-structured interviews with third and fourth year medical students at one institution. The interviews aimed to foster rich discussion about students' feedback experiences with residents during clinical rotations. Data collection and analysis proceeded iteratively. Initial interviews were independently open-coded by three investigators and then collaboratively refined. Codes were applied to subsequent interviews, and new codes were developed. During the final stages of analysis, we organised our themes by drawing on a sociocultural perspective to examine students' perceptions of relationship-building with residents and when and how this influenced feedback and learning. RESULTS: From the students' perspectives, when residents contributed to building interpersonal relationships with students, this in turn influenced students' receptivity to both encouraging and constructive feedback conversations. In the context of resident-student relationships that were perceived as supportive, resident feedback influenced how students approached learning and working in the clinical environment, as well as students' visions of their future selves. In unsupportive relationships, students were less inclined to engage in feedback with residents and students noted resident behaviours that they wanted to avoid in themselves. CONCLUSION: Residents are uniquely positioned to create a strong educational alliance with students in which feedback conversations can flourish. Focusing educational efforts on resident feedback conversations has the potential to significantly impact the feedback culture of our clinical environments.

Examining the validity argument for the Ottawa Surgical Competency Operating Room Evaluation (OSCORE): a systematic review and narrative synthesis.

The Ottawa Surgical Competency Operating Room Evaluation (OSCORE) is an assessment tool that has gained prominence in postgraduate competency-based training programs. We undertook a systematic review and narrative synthesis to articulate the underlying validity argument in support of this tool. Although originally developed to assess readiness for independent performance of a procedure, contemporary implementation includes using the OSCORE for entrustment supervision decisions. We used systematic review methodology to search, identify, appraise and abstract relevant articles from 2005 to September 2020, across MEDLINE, EMBASE and Google Scholar databases. Nineteen original, English-language, quantitative or qualitative articles addressing the use of the OSCORE for health professionals' assessment were included. We organized and synthesized the validity evidence according to Kane's framework, articulating the validity argument and identifying evidence gaps. We demonstrate a reasonable validity argument for the OSCORE in surgical specialties, based on assessing surgical competence as readiness for independent performance for a given procedure, which relates to ad hoc, retrospective, entrustment supervision decisions. The scoring, generalization and extrapolation inferences are well-supported. However, there is a notable lack of implications evidence focused on the impact of the OSCORE on summative decision-making within surgical training programs. In non-surgical specialties, the interpretation/use argument for the OSCORE has not been clearly articulated. The OSCORE has been reduced to a single-item global rating scale, and there is limited validity evidence to support its use in workplace-based assessment. Widespread adoption of the OSCORE must be informed by concurrent data collection in more diverse settings and specialties.

A systematic review of evidence-based practices for clinical education and health care delivery in the clinical teaching unit.

BACKGROUND: The clinical teaching unit is a widespread clinical training model that requires reform to prepare physicians for practice in the 21st century. In this systematic review, we aimed to identify evidence-based practices in internal medicine clinical teaching units that contribute to improved clinical education and health care delivery. METHODS: We searched several databases from 1993 until Apr. 5, 2021, to identify published studies in inpatient clinical teaching units that involved medical trainees and reported outcomes related to trainee education or health care delivery. We identified emergent themes using a narrative approach and determined confidence in review findings using the Grading of Recommendations Assessment, Development and Evaluation Confidence in the Evidence from Reviews of Qualitative Research (GRADE-CERQual) methodology. RESULTS: We included 107 studies of internal medicine clinical teaching units, of which 93 (87%) were conducted in North America. Surveys (n = 31, 29%), trials (n = 17, 16%) and narrative studies (n = 15, 14%) were the most prevalent study designs. Practices identified as contributing to improved clinical education or health care delivery included purposeful rounding (high confidence), bedside rounding (moderate confidence), resource stewardship interventions (high confidence), interprofessional rounds (moderate confidence), geographic wards (moderate confidence), allocating more trainee time to patient care or educational activities (moderate confidence), "drip" continuous models of admission (moderate confidence), limiting duty hours (moderate confidence) and limiting clinical workload (moderate confidence). INTERPRETATION: In this review, we identified several evidence-based practices that may contribute to improved educational and health care outcomes in clinical teaching unit settings. These findings may offer guidance for policies, resource allocation and staffing of teaching hospitals.

Numbers Encapsulate, Words Elaborate: Toward the Best Use of Comments for Assessment and Feedback on Entrustment Ratings.

The adoption of entrustment ratings in medical education is based on a seemingly simple premise: to align workplace-based supervision with resident assessment. Yet it has been difficult to operationalize this concept. Entrustment rating forms combine numeric scales with comments and are embedded in a programmatic assessment framework, which encourages the collection of a large quantity of data. The implicit assumption that more is better has led to an untamable volume of data that competency committees must grapple with. In this article, the authors explore the roles of numbers and words on entrustment rating forms, focusing on the intended and optimal use(s) of each, with a focus on the words. They also unpack the problematic issue of dual-purposing words for both assessment and feedback. Words have enormous potential to elaborate, to contextualize, and to instruct; to realize this potential, educators must be crystal clear about their use. The authors set forth a number of possible ways to reconcile these tensions by more explicitly aligning words to purpose. For example, educators could focus written comments solely on assessment; create assessment encounters distinct from feedback encounters; or use different words collected from the same encounter to serve distinct feedback and assessment purposes. Finally, the authors address the tyranny of documentation created by programmatic assessment and urge caution in yielding to the temptation to reduce words to numbers to make them manageable. Instead, they encourage educators to preserve some educational encounters purely for feedback, and to consider that not all words need to become data.

Entrustable Professional Activities and Entrustment Decision Making: A Development and Research Agenda for the Next Decade.

To establish a research and development agenda for Entrustable Professional Activities (EPAs) for the coming decade, the authors, all active in this area of investigation, reviewed recent research papers, seeking recommendations for future research. They pooled their knowledge and experience to identify 3 levels of potential research and development: the micro level of learning and teaching; the meso level of institutions, programs, and specialty domains; and the macro level of regional, national, and international dynamics. Within these levels, the authors categorized their recommendations for research and development. The authors identified 14 discrete themes, each including multiple questions or issues for potential exploration, that range from foundational and conceptual to practical. Much research to date has focused on a variety of issues regarding development and early implementation of EPAs. Future research should focus on large-scale implementation of EPAs to support competency-based medical education (CBME) and on its consequences at the 3 levels. In addition, emerging from the implementation phase, the authors call for rigorous studies focusing on conceptual issues. These issues include the nature of entrustment decisions and their relationship with education and learner progress and the use of EPAs across boundaries of training phases, disciplines and professions, including continuing professional development. International studies evaluating the value of EPAs across countries are another important consideration. Future studies should also remain alert for unintended consequences of the use of EPAs. EPAs were conceptualized to support CBME in its endeavor to improve outcomes of education and patient care, prompting creation of this agenda.

A collective case study of supervision and competence judgments on the inpatient internal medicine ward.

INTRODUCTION: Workplace-based assessment in competency-based medical education employs entrustment-supervision scales to suggest trainee competence. However, clinical supervision involves many factors and entrustment decision-making likely reflects more than trainee competence. We do not fully understand how a supervisor's impression of trainee competence is reflected in their provision of clinical support. We must better understand this relationship to know whether documenting level of supervision truly reflects trainee competence. METHODS: We undertook a collective case study of supervisor-trainee dyads consisting of attending internal medicine physicians and senior residents working on clinical teaching unit inpatient wards. We conducted field observations of typical daily activities and semi-structured interviews. Data was analysed within each dyad and compared across dyads to identify supervisory behaviours, what triggered the behaviours, and how they related to judgments of trainee competence. RESULTS: Ten attending physician-senior resident dyads participated in the study. We identified eight distinct supervisory behaviours. The behaviours were enacted in response to trainee and non-trainee factors. Supervisory behaviours corresponded with varying assessments of trainee competence, even within a dyad. A change in the attending's judgment of the resident's competence did not always correspond with a change in subsequent observable supervisory behaviours. DISCUSSION: There was no consistent relationship between a trigger for supervision, the judgment of trainee competence, and subsequent supervisory behaviour. This has direct implications for entrustment assessments tying competence to supervisory behaviours, because supervision is complex. Workplace-based assessments that capture narrative data including the rationale for supervisory behaviours may lead to deeper insights than numeric entrustment ratings.

Learning Curves in Health Professions Education Simulation Research: A Systematic Review.

SUMMARY STATEMENT: Learning curves are used in health professions education to graphically represent paths to competence and expertise. However, research using learning curves often omit important information. The authors conducted a systematic review of the reporting quality of learning curves in simulation-based education research to identify specific areas for improvement. Reviewers extracted information on graphical, statistical, and conceptual elements. The authors identified 230 eligible articles. Most learning curve elements were reported infrequently, including use of an optimal linking function, detailed description of feedback or learning intervention, use of advanced visualization techniques such as overlaying and stacking, and depiction of competency thresholds. Reporting did not improve over time for most elements. Reporting of learning curves in health professions education research is incomplete and often underutilizes their desirable properties. Recommendations for improvement of the statistical, graphical, and conceptual reporting of learning curves, as well as applications to simulation research and education, are presented.

Learning With Patients, Students, and Peers: Continuing Professional Development in the Solo Practitioner Workplace.

Conceptualizations of workplace learning have moved from knowledge acquisition to learning as participation in the practices and cultures of the workplace environment. Along with this has come an appreciation of applicability of sociocultural learning theories, which frame learning as occurring within "communities of practice" or learning being "situated" within a workplace environment where collaboration and social interaction are fundamental to the learning process. These conceptualizations of workplace learning are ideally suited to health professions where learners are supervised in clinical work environments and then continue to work in team-based environments as graduates. However, what happens to workplace learning for novice practitioners who have limited periods of clinical supervision and then graduate into solo or small group practices (which may also be in rural or remote locations) and embark on long working careers without supervision? This paper argues workplace learning needs to be scaffolded and supported to reach its full potential in these environments. Drawing on workplace-based learning theory, we highlight the ubiquitous nature of learning in the workplace, the importance of active engagement, reflection, and individual meaning making. Through this reframing of traditional notions of continuing professional development, we emphasize the importance of patients, students, and other practitioners as partners in workplace learning for solo practitioners. We also focus on the role of educators, professional associations, and regulators in helping solo practitioners recognize, access, and maximize the learning opportunities inherent in relatively isolated practice environments.

The nature of learning from simulation: Now I know it, now I'll do it, I'll work on that.

CONTEXT: Ongoing learning in complex clinical environments requires health professionals to assess their own performance, manage their learning, and modify their practices based on self-monitored progress. Self-regulated learning (SRL) theory suggests that although learners may be capable of such learning, they often need guidance to enact it effectively. Debriefings following simulation may be an ideal time to support learners' use of SRL in targeted areas, but the extent to which they are optimally fostering these practices has not been examined. METHODS: A qualitative study informed by grounded theory methodology was conducted in the context of three interprofessional in situ trauma simulations at our level 1 trauma centre. A total of 18 participants were interviewed both immediately and 5-6 weeks after the simulation experience. Transcripts were analysed using an iterative constant comparative approach to explore concepts and themes regarding the nature of learning from and after simulation. RESULTS: During initial interviews, there were many examples of acquired content knowledge and straightforward practice changes that might not require ongoing SRL to enact well in practice. However, even for skills identified as needing to be 'worked on,' SRL strategies were lacking. At follow-up interviews, some participants had evolved more specific learning goals and rudimentary plans for implementation and improvement, but suggested this was prompted by the study interview questions rather than the simulation debriefing itself. CONCLUSIONS: Overall, participants did not engage in fulsome development of SRL plans based on the simulation and debriefing; however, there were elements of SRL present, particularly after participants were given time to reflect on the interview questions and their own goals. This suggests that simulation training can support the use of SRL. However, debriefing approaches might be better optimised to take full advantage of the opportunity to encourage and foster SRL in practice after the simulation is over.

Virtual Dissection with Clinical Radiology Cases Provides Educational Value to First Year Medical Students.

RATIONALE AND AIM: In virtual dissection, three-dimensional computed tomography scans are viewed on a near-life size virtual dissection table and through touchscreen technology, students work together to manipulate the data to perform their dissection. The purpose of this study was to develop a Virtual Dissection Curriculum for first year medical students and to assess its educational value as well as students' preferred pedagogy for learning with this new technology. METHODS: One hundred and five first-year medical students participated in a case-based virtual dissection curriculum and were invited to complete a theory-based post experience survey. Eight unique clinical cases were selected based on the first-year curricular objectives and divided into four 30-minute sessions. In groups of 6-8, students reviewed the cases with a radiologist. First, students' reactions to virtual dissection were measured by three constructs using a 5-point Likert scale: quality of curriculum design (11 questions), impact on learning (7 questions), and comfort with technology (3 questions). Second, students ranked the usefulness of six pedagogical approaches for this technology. Responses were tabulated and rank order item lists were generated statistically using the Schulze method where appropriate. RESULTS: The survey response rate was 83% (87/105). Overall, students' reactions to virtual dissection were positive across all three measured constructs. Most students indicated that the cases were of an appropriate level of difficulty (90%) and that virtual dissection improved their understanding of disease and pathology (89%), the clinical relevance of anatomy (77%), and visuospatial relationships (64%). Almost all students (94%) reported that the curriculum improved understanding of the role of the radiologist in patient care. Students felt that the "very useful" pedagogical approaches were small group demonstration (68%) and problem-based learning (51%). CONCLUSION: First-year medical students perceive the use of virtual dissection as a valuable tool for learning anatomy and radiology. This technology enables the integration of clinical cases and radiology content into preclinical learning.

Learning After the Simulation Is Over: The Role of Simulation in Supporting Ongoing Self-Regulated Learning in Practice.

The complex and dynamic nature of the clinical environment often requires health professionals to assess their own performance, manage their learning, and modify their practices based on self-monitored progress. Self-regulated learning studies suggest that while learners may be capable of such in situ learning, they often need guidance to enact it effectively. In this Perspective, the authors argue that simulation training may be an ideal venue to prepare learners for self-regulated learning in the clinical setting but may not currently be optimally fostering self-regulated learning practices. They point out that current simulation debriefing models emphasize the need to synthesize a set of identified goals for practice change (what behaviors might be modified) but do not address how learners might self-monitor the success of their implementation efforts and modify their learning plans based on this monitoring when back in the clinical setting. The authors describe the current models of simulation-based learning implied in the simulation literature and suggest potential targets in the simulation training process, which might be optimized to allow medical educators to take full advantage of the opportunity simulation provides to support and promote ongoing self-regulated learning in practice.

Necessary but not sufficient: identifying conditions for effective feedback during internal medicine residents' clinical education.

Competency-based medical education and programmatic assessment intend to increase the opportunities for meaningful feedback, yet these conversations remain elusive. By comparing resident and faculty perceptions of feedback opportunities within one internal medicine residency training program, we sought to understand whether and how principles underlying meaningful feedback could be supported or constrained across a variety of feedback opportunities. Using case-study qualitative methodology, interviews and focus groups were conducted to explore 19 internal medicine residents' and 7 faculty members' perceptions of feedback across a variety of feedback opportunities: coaching, mini-CEXs, in-training evaluation reports and routine clinical supervision. Our data analysis moved iteratively between developing conceptual understandings and fine-grained analyses, while attending to both deductive and inductive analysis. Our results suggest that all feedback opportunities, including those created through formalized assessments, can foster meaningful feedback if faculty establish a trusting relationship with the resident, base their feedback on direct observation and support resident learning. However, formalized assessments were often perceived as inhibiting the conditions for meaningful feedback. A coaching program provided a context in which meaningful feedback could arise, in part because faculty were supported in shifting their focus from patient to resident. Meaningful feedback in clinical education may be fostered across a variety of feedback opportunities, however, it is often constrained by assessment. We must consider whether increasing the frequency of formative assessments may inhibit efforts to improve our feedback cultures while, in contrast, freeing up faculty to focus on supporting resident learning could improve these cultures.

Integrated virtual and cadaveric dissection laboratories enhance first year medical students' anatomy experience: a pilot study.

BACKGROUND: Radiology integration into medical anatomy courses is well established, but there is a paucity of literature on integrating virtual dissection into cadaveric dissection laboratories. Virtual dissection is the digital dissection of medical images on touchscreen anatomy visualization tables. The purpose of this pilot study was to investigate the feasibility of integrating virtual dissection into a first-year medical cadaver-based anatomy course and to assess students' overall attitude towards this new technology. METHODS: All students in first-year medicine at a single medical school participated in this study (n = 292). Six virtual dissection laboratories, which focused on normal anatomy, were developed and integrated into a cadaver-based anatomy course. The virtual dissection table (VDT) was also integrated into the final anatomy spot exam. Following the course, students completed a short evidence-informed survey which was developed using a theoretical framework for curriculum evaluation. Numerical data were tabulated, and qualitative content analysis was performed on students' unstructured comments. RESULTS: The survey response rate was 69.2% (n = 202/292). Most (78.7%) students reported that virtual dissection enhanced their understanding of the cadaveric anatomy and the clinical applications of anatomy. Most (73.8%) students also felt that the VDT was an effective use of the laboratory time. Thirteen narrative comments were collected, most of which (61.5%) identified strengths of the curriculum. CONCLUSIONS: In this pilot study, students perceived that their learning was enhanced when virtual dissection was combined with a cadaver-based anatomy laboratory. This study demonstrates that there is potential for virtual dissection to augment cadaveric dissection in medical education.